Electronic apparatus, method for controlling thereof and the computer readable recording medium

ABSTRACT

An electronic apparatus is disclosed. The electronic apparatus includes an inputter configured to receive a binocular image which is a captured image of both eyes of a user and a stereo image which is an image of a direction corresponding to a gaze of the user captured at locations spaced apart from each other, and a processor configured to detect a watch point of a user in the stereo image by using the binocular image, obtain a disparity map in the input stereo image, and compensate the detected watch point using the obtained disparity map.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority from Korean Patent Application No.10-2017-0113647, filed on Sep. 6, 2017 in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference in its entirety.

BACKGROUND Field

Aspects of the embodiments relate to an electronic apparatus, acontrolling method thereof, and a computer readable recording medium,and more particularly, to an electronic apparatus for detecting a watchpoint of a user in a three-dimensional space, a controlling methodthereof, and a computer readable recording medium.

Description of the Related Art

The trend of smartphones and tablet PCs that have swept the world forseveral years has changed from consumption tendency to lifestylepatterns. That much, IT apparatuses are in close relationship with ourdaily lives these days. This is because of that new technologies whichcould be seen in a movie before can be easily used by anyone now. One ofthe representative new technologies is an ‘Augmented Reality (AR)’.

Detecting an object gazed by a user in a three-dimensional space fromthe user's gaze is a basic interface for Human-computer interaction(HCI) in the system such as a Head-up display (HUD) and the like.

The technology for tracking a user's gaze in a three-dimensional spacehas been actively studied as an important element technology for anobject recognition and the HCI. Tracking the user's gaze in athree-dimensional space has been basically performed based on theobservation of user's eyes and the result thereof. However, only withthe binocular observation result, there is a limitation in accuratelyfinding the watch point in three-dimensional space at a distance as alimitation of a measurement accuracy.

Thus, recently, in order to improve the tracking performance and toutilize the results thereof for an object recognition, a computerinteraction, etc., a method of additionally using a camera for capturingthe same gaze direction as the user has been actively studied.Specifically, it is necessary to perform a precise matching processbetween the binocular gaze vector which is derived by observing botheyes and the image of the user's gaze direction. Even if a precisematching is performed in advance, there is a problem that it isdifficult to accurately detect the watch point in a three-dimensionalspace because of a binocular visual vector error, a match error, etc.

Accordingly, in order to accurately detect the watch point in athree-dimensional space of a user, a need for the technology forcompensating the error in gaze vector has been raised.

SUMMARY

An aspect of the embodiment has been made to address the problemsdescribed above and to provide an electronic apparatus for accuratelydetecting a watch point of a user in a three-dimensional space using adisparity map, a controlling method thereof, and a computer readablerecording medium.

According to an embodiment, there is provided an electronic apparatusincluding an inputter configured to receive a binocular image which is acaptured image of both eyes of a user and a stereo image which is animage of a direction corresponding to a gaze of the user captured atlocations spaced apart from each other, and a processor configured todetect a watch point of a user in the stereo image by using thebinocular image, obtain a disparity map in the input stereo image, andcompensate the detected watch point using the obtained disparity map.

In this case, the stereo image may include a first image correspondingto a left eye of a user and a second image corresponding to a right eyeof a user, and the processor may detect a left eye watch point of a userin the first image and a right eye watch point in the second image byusing the binocular image, obtain a first disparity map and a seconddisparity map by performing stereo matching of the first image and thesecond image, and compensate the detected left eye watch point and thedetected right eye watch point by using the first disparity map and thesecond disparity map.

In this case, the processor may compensate the detected left eye watchpoint and the detected right eye watch point using the first disparitymap and the second disparity map to minimize a difference between adisparity of the left eye watch point and a disparity of the right eyewatch point.

The processor may compensate the detected left eye watch point and thedetected right eye watch point to minimize a difference between alocation of the left eye watch point and a location of the right eyewatch point in a state in which horizontal deviation of the first imageand the second image is compensated.

The processor may compensate vertical deviation of the first image andthe second image, and perform stereo matching using the compensatedfirst image and the compensated second image.

The processor may obtain a three dimensional watch point using thecompensated left eye watch point and the compensated right eye watchpoint.

The processor may obtain vector and distance information of a binoculargaze using the compensated left eye watch point and the compensatedright eye watch point.

The inputter may include an infrared camera configured to captures botheyes of the user, and a stereo camera configured to captures a directioncorresponding to a gaze of the user at locations spaced apart from eachother.

According to an embodiment, there is provided a method for controllingan electronic apparatus including receiving a binocular image which is acaptured image of both eyes of a user and a stereo image which is animage of a direction corresponding to a gaze of the user captured atlocations spaced apart from each other, detecting a watch point of auser in the stereo image by using the binocular image, obtaining adisparity map in the input stereo image, and compensating the detectedwatch point using the obtained disparity map.

In this case, the stereo image may include a first image correspondingto a left eye of a user and a second image corresponding to a right eyeof a user, the detecting the watch point may include detecting a lefteye watch point of a user in the first image and a right eye watch pointin the second image by using the binocular image, the obtaining mayinclude obtaining a first disparity map and a second disparity map byperforming stereo matching of the first image and the second image, andthe compensating may include compensating the detected left eye watchpoint and the detected right eye watch point by using the firstdisparity map and the second disparity map.

The compensating may include compensating the detected left eye watchpoint and the detected right eye watch point using the first disparitymap and the second disparity map to minimize a difference between adisparity of the left eye watch point and a disparity of the right eyewatch point.

The compensating may include compensating the detected left eye watchpoint and the detected right eye watch point to minimize a differencebetween a location of the left eye watch point and a location of theright eye watch point in a state in which horizontal deviation of thefirst image and the second image is compensated.

The controlling method may further include compensating verticaldeviation of the first image and the second image, and the obtaining mayinclude performing stereo matching using the compensated first image andthe compensated second image.

The controlling method may further include obtaining a three dimensionalwatch point using the compensated left eye watch point and thecompensated right eye watch point.

The controlling method may further include obtaining vector and distanceinformation of a binocular gaze using the compensated left eye watchpoint and the compensated right eye watch point.

The receiving may include capturing both eyes of the user using aninfrared camera, and capturing a direction corresponding to a gaze ofthe user at locations spaced apart from each other.

According to an embodiment, there is provided a computer readablerecording medium which includes a program to execute a method forcontrolling an electronic apparatus including receiving a binocularimage which is a captured image of both eyes of a user and a stereoimage which is an image of a direction corresponding to a gaze of theuser captured at locations spaced apart from each other, detecting awatch point of a user in the stereo image by using the binocular image,obtaining a disparity map in the input stereo image, and compensatingthe detected watch point using the obtained disparity map.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view illustrating a system for detecting a watch point of auser in a three-dimensional space according to an embodiment;

FIG. 2 is a block diagram illustrating a brief configuration of anelectronic apparatus according to an embodiment;

FIG. 3 is a block diagram illustrating a specific configuration of anelectronic apparatus of FIG. 2;

FIG. 4 is a view illustrating a disparity map according to anembodiment;

FIG. 5 is a view illustrating a method for compensating a watch pointaccording to an embodiment;

FIG. 6 is a view illustrating a watch point detected in a stereo image;

FIG. 7 is a view illustrating a watch point obtained by compensating awatch point of FIG. 6 according to an embodiment; and

FIG. 8 is a flow chart illustrating a method for compensating a watchpoint according to an embodiment.

DETAILED DESCRIPTION

Terms used in the present disclosure will be briefly explained, andembodiments will be described in great detail.

Terms used in the present disclosure are selected as generalterminologies currently widely used in consideration of theconfiguration and functions of the present disclosure, but can bedifferent depending on intention of those skilled in the art, aprecedent, appearance of new technologies, and the like. Further, inspecific cases, terms may be arbitrarily selected. In this case, themeaning of the terms will be described in the description of thecorresponding embodiments. Accordingly, the terms used in thedescription should not necessarily be construed as simple names of theterms, but be defined based on meanings of the terms and overallcontents of the present disclosure.

The embodiments may vary, and may be provided in different exampleembodiments. Various embodiments will be described with reference toaccompanying drawings. However, this is not intended to limit the scopeto an embodiment, and therefore, it should be understood that all themodifications, equivalents or substitutes included under the inventedspirit and technical scope are encompassed. In describing theembodiments, well-known functions or constructions are not described indetail since they would obscure the specification with unnecessarydetail.

The terms such as “first,” “second,” and so on may be used to describe avariety of elements, but the elements should not be limited by theseterms. The terms are used simply to distinguish one element from otherelements.

Singular forms are intended to include plural forms unless the contextclearly indicates otherwise. In the embodiments, the terms “include” and“comprise” designate the presence of features, numbers, steps,operations, components, elements, or a combination thereof that arewritten in the specification, but do not exclude the presence orpossibility of addition of one or more other features, numbers, steps,operations, components, elements, or a combination thereof.

In an embodiment, ‘a module’ or ‘a unit’ performs at least one functionor operation, and may be realized as hardware, software, or combinationthereof. In addition, a plurality of ‘modules’ or ‘units’ may beintegrated into at least one module and may be realized as at least oneprocessor in an integrated manner except for ‘modules’ or ‘units’ thatshould be realized in specific hardware.

Hereinafter certain embodiments will now be described in greater detailwith reference to the accompanying drawings to enable those skilled inthe art to work the same with ease. However, embodiments may be realizedin a variety of different configurations, and not limited todescriptions provided herein. Further, those that are irrelevant withthe description are omitted so as to describe exemplary embodiments moreclearly, and similar drawing reference numerals are used for the similarelements throughout the description.

Below, embodiments will be described in detail with reference to theattached drawings.

FIG. 1 is a view illustrating a system for detecting a watch point of auser in a three-dimensional space according to an embodiment.

Referring to FIG. 1, the system to detect a watch point of a user 10according to an embodiment may include a plurality of cameras 111 and112 that capture both eyes of the user 10 and a plurality of cameras 113and 114 that capture a direction to which the user 10 is watching.

Specifically, the plurality of cameras 111 and 112 that capture botheyes of the user 10 may include the camera 111 that captures a left eyeof the user 10 and the camera 112 that captures a right eye of the user10. Here, the plurality of cameras 111 and 112 that capture both eyes ofthe user 10 may be for detecting the gaze of the user 10. Here, theplurality of cameras 111 and 112 that capture both eyes of the user 10may be an infrared camera or a general camera using visible light.

Meanwhile, FIG. 1 illustrates that the plurality of cameras capture botheyes of the user 10 respectively, but in the implementation, one cameramay capture both eyes of a user and in the image of the both eyes, aleft eye and a right eye of the user 10 may be detected respectively.

Meanwhile, the plurality of cameras 113 and 114 that capture a directionwhere the user 10 is watching may include the camera 113 that capturesthe direction where a left eye of the user is watching and the camera114 that captures the direction where a right eye of the user 10 iswatching. Here, the plurality of cameras 113 and 114 that capture thedirection where the user 10 is watching is for capturing an image of thearea where the user 10 is presumed to be watching. Here, the pluralityof cameras 113 and 114 that capture the direction where the user iswatching may be stereo cameras to obtain a stereoscopic image.

Meanwhile, in the system to detect a watch point of a user in athree-dimensional space, the watch point of a user in a stereo image maybe detected using a binocular image of the user 10 captured by theplurality of cameras 111 and 112 that capture the both eyes of the user10 and using a stereo image captured by the plurality of cameras 113 and114 that capture the direction being gazed by the user 10, and thedetected watch point may be compensated using a disparity map obtainedwith the stereo image. Meanwhile, a method for compensating the detectedwatch point will be described in detail with reference to FIGS. 2 to 8below.

FIG. 2 is a block diagram illustrating a brief configuration of anelectronic apparatus according to an embodiment.

Referring to FIG. 2, the electronic apparatus 100 may include aninputter 110 and a processor 120.

The inputter 110 may receive a binocular image which is an image of botheyes of a user. Here, the inputter 110 may receive a plurality of imagesin which both eyes of the user are captured respectively, or receive oneimage in which both eyes of the user are captured at the same time.

Specifically, the inputter 110 may include at least one camera tocapture both eyes of a user. Here, the inputter 110 may include at leastone camera when capturing both eyes of the user at the same time, andwhen capturing both eyes respectively, the inputter 110 may include atleast two or more than two cameras. Here, the inputter 110 may captureboth eyes of the user with an infrared camera or a general camera usingvisible light.

In addition, the inputter 110 may receive a stereo image which is animage of the direction corresponding to the gaze of the user, capturedat locations spaced apart from each other.

Specifically, the inputter 110 may include a camera for capturing adirection corresponding to the gaze of a left eye of a user and a camerafor capturing a direction corresponding to the gaze of a right eye ofthe user. Here, a plurality of images in which each directioncorresponding to the user's gaze is captured respectively may be calleda stereo image. Here, the stereo image may include a left eye imagecorresponding to a left eye of a user and a right eye imagecorresponding to a right eye of a user. In addition, practically thereis a difference between the area where the left eye gazes and the areawhere the right eye gazes because of the distance between the left eyeand the right eye of the user, and thus the first image and the secondimage may be generated so that there is a difference in the locations ofthe corresponding points in each image.

Meanwhile, if a camera is not included in the electronic apparatus 100,the inputter 110 may include a communicator (not illustrated) thatreceives a binocular image and a stereo image from an external camera.

The processor 120 may detect a watch point of a user in a stereo imageusing the binocular image received by the inputter 110. Specifically,the processor 120 may detect a left eye watch point watched by the lefteye in the left eye image corresponding to the left eye by using a gazevector of the left eye detected in the received binocular image. Inaddition, the processor 120 may detect a right eye watch point watchedby the right eye in the right eye image corresponding to the right eyeusing a gaze vector of the right eye detected in the received binocularimage.

In addition, the processor 120 may generate a disparity map using astereo image. Specifically, the processor 120 may generate the disparitymap regarding each of the left eye image and the right eye image.Specifically, the processor 120 may perform stereo matching of the lefteye image and the right eye image and generate the disparity mapregarding each image. Here, the stereo matching may mean that distanceinformation is extracted using the degree of difference between thecorresponding points in two images in different watch point.

This reflects that there is a difference in the areas watched by a lefteye and a right eye according to the distance between the left eye andthe right eye, and the map is generated by using that if the same objectis near in an image, there is a big difference between the locations ofthe object in two images, and if the distance between the same object isfar, there is a small difference between the locations of the object intwo images. As such, the distance information between the location ofthe object and the location in which a camera is located in the imagemay be known by using the disparity map.

Here, the processor 120 may compensate the vertical deviation of theleft eye image and the right eye image which are input stereo images.According thereto, there is only a left-right deviation in the locationof the corresponding points in the left eye image and the right eyeimage in which the vertical deviation is compensated, further accuratedetection of a watch point become possible. In addition, the processor120 performs stereo matching of the compensated left eye image and righteye image and generates the disparity map regarding each of the left eyeimage and the right eye image.

The processor 120 may compensate the detected watch point using thegenerated disparity map. Specifically, the processor 120 may compensatethe location of the left eye watch point and the location of the righteye watch point using the disparity map generated regarding the left eyeimage and the disparity map generated regarding the right eye image.

Specifically, the processor 120 may calculate the range in which thecompensation may be performed regarding each watch point based on thedetected left eye watch point and right eye watch point, and compensatethe left eye watch point and right eye watch point within the calculatedrange.

Specifically, the processor 120 may compensate the location of thedetected left eye watch point and the location of the right eye watchpoint so that the difference between the location of the left eye watchpoint and the location of the right eye watch point to be minimized.Specifically, in the state in which a horizontal deviation of the lefteye image and right eye image is compensated, the processor 120 maycompensate the locations of the detected left eye watch point anddetected right eye watch point so that the difference between thelocations of the left eye watch point in the left eye image in which thehorizontal deviation is compensated and the location of the right eyewatch point in the right eye image in which the horizontal deviation iscompensated, is minimized. Meanwhile, in the state in which thehorizontal deviation of the left eye image and right eye image is notcompensated, the processor 120 may compensate the location of thedetected left eye watch point and the location of the detected right eyewatch point to be converged to the difference in the locations of thecorresponding points in the left eye image and the right eye image.

In addition, the processor 120 may compensate the locations of thedetected left watch point and the detected right watch point by usingthe disparity map generated regarding the left eye image and thedisparity map generated regarding the right eye image so the differencein the disparity of the left watch point and the right watch point to beminimized. This is because of that even if the difference of distance isnot big in a two-dimensional image, if watch point is detected ondifferent areas respectively based on the boundary of an object, it isdifficult to determine the watch point as a watch point that a user iswatching.

Meanwhile, the processor 120 may select an optimum compensated left eyewatch point ({circumflex over (x)}′₁) and an optimum compensated righteye watch point ({circumflex over (x)}′₂) that satisfy the followingformula (1) in which the above described compensation method isreflected within the calculated compensable area.

$\begin{matrix}{{{\underset{\text{?}\text{?}}{argmin}\left\lbrack {\left\{ {{d\left( {\text{?}\text{?}} \right)}^{2} + \left( {\text{?},\text{?}} \right)^{2}} \right\} + {\lambda \left\{ {\left( {{M_{1}\left( {\hat{x}}_{1}^{\prime} \right)} - {M_{2}\left( {\hat{x}}_{2}^{\prime} \right)}} \right)^{2} + \left( {{M_{1}\left( {\hat{x}}_{1}^{\prime} \right)} - {d\left( {\text{?},{\hat{x}}_{2}^{\prime}} \right)}} \right)^{2} + \left( {{M_{2}\left( {\hat{x}}_{2}^{\prime} \right)} - {d\left( {\text{?}\text{?}} \right)}} \right)^{2}} \right\}}} \right\rbrack}{\text{?}\text{indicates text missing or illegible when filed}}}\mspace{304mu}} & (1)\end{matrix}$

Here, x1′ and x2′ may mean the locations of the left eye watch point andthe right eye watch point detected from each stereo image in whichvertical deviation is compensated, M1(x) and M2(x) may mean disparityvalues in a specific location from each stereo image in which verticaldeviation is compensated, and d({circumflex over (x)}′₁,{circumflex over(x)}′₂) may mean a relative coordinate difference between thecompensated left eye watch point ({circumflex over (x)}′₁) and thecompensated right eye watch point ({circumflex over (x)}′₂) in thestereo image in which a vertical deviation is compensated.

Referring to formula (1), the processor 120 may select {circumflex over(x)}′₁ and {circumflex over (x)}′₂ so that a location change of thedetected watch point and the compensated watch point is minimum, and thedisparity difference of the watch point in each disparity map isminimum.

In addition, the processor 120 may select {circumflex over (x)}′₁ and{circumflex over (x)}′₂ so that each of the disparity of the compensatedleft eye watch point and the disparity of the compensated right eyewatch point, and a relative coordinate difference between thecompensated left eye watch point ({circumflex over (x)}′₁) and thecompensated right eye watch point) ({circumflex over (x)}′₂) to beminimum. This is because of that the disparity of the watch point iscalculated by the difference between each corresponding point in thestereo image.

In addition, the processor 120 may generate a three-dimensional watchpoint using the compensated left eye watch point and the compensatedright eye watch point. Specifically, since the left eye watch point andthe right eye watch point primarily detected in the stereo image are notmatched, a three-dimensional watch point could not be generated orgenerated in an inaccurate location, but by using the compensated lefteye watch point and the compensated right eye watch point, furtheraccurate three-dimensional watch point may be generated.

In addition, the processor 120 may generate a vector and distanceinformation of a binocular gaze using the compensated left eye watchpoint and the compensated right eye watch point. Specifically, theprocessor 120 may generate the vector and distance information of thebinocular gaze regarding the generated three-dimensional watch point.

As described above, according to compensating the left eye watch pointand the right eye watch point using the disparity map, further accuratewatch point of a user in a three-dimensional space may be detected.

FIG. 3 is a block diagram illustrating a specific configuration of anelectronic apparatus of FIG. 2.

Referring to FIG. 3, the electronic apparatus 100 may include theinputter 110, the processor 120, the memory 130, the communicator 140,the display 150 and a video processor 160. Here, some operations of theinputter 110 and the processor 120 are the same as the configurationdescribed in FIG. 2, and thus overlapping descriptions will be omitted.

The processor 120 may include a random access memory (RAM) 121, a readonly memory (ROM) 122, a central processing unit (CPU) 123, a graphicprocessing unit (GPU) 124, and a bus 125. The RAM 121, the ROM 122, theCPU 123, the GPU 124, and so on may be connected to each other throughthe bus 125.

The CPU 123 accesses the memory 140 to perform booting using the 0/Sstored in the memory 140. The CPU 123 may perform various operations byusing the various programs, contents, data, and the like stored in thememory 140.

The ROM 122 stores a set of instructions for system booting. When aturn-on command is input and thus the power is supplied, the CPU 123 maycopy the 0/S stored in the memory 180 to RAM 121 according to theinstructions stored in the ROM 122, and boot the system by executing the0/S. When the booting is completed, the CPU 123 may copy the variousprograms stored in the memory 140 to the RAM 121, and perform variousoperations by implementing the programs copied to the RAM 121.

The GPU 124 may display a UI on the display 170 when the booting of theelectronic apparatus 100 is completed. In detail, the GPU 124 maygenerate a screen including various objects such as an icon, an image, atext, etc. using a calculator (not shown) and a renderer (not shown).The calculator may calculate attribute values such as coordinates,shape, size, and color of each object to be displayed according to thelayout of the screen. The renderer may generate screens of variouslayouts including objects based on the attribute value calculated in theoperation unit. The screens (or user interface window) generated in therenderer may be provided to the display, and displayed on each of a maindisplay area and a sub display area. Meanwhile, it has been describedthat the image processing operation according to an embodiment isperformed by the GPU 124, but in the implementation, the operation maybe performed by the CPU 123 or the GPU 124.

The memory 130 may be realized as a recording medium in the electronicapparatus 100 or as an external storage medium, such as a removable diskincluding a USB memory, a web server via network, and the like.Specifically, the memory 130 may include a hard disk, SSD, a memorycard, a ROM, a USB memory and the like.

Specifically, the memory 130 may store the information on the binocularwatch point detected by using the binocular image and a stereo imageinput through the inputter 110. In addition, the memory 130 may storethe disparity map separately generated with regard to the stereo image.In addition, the memory 130 may store the information regarding thecompensated binocular watch point using the disparity map by theprocessor 120.

In addition, the memory 130 may store the information on thethree-dimensional watch point generated by using the compensatedbinocular watch point. Specifically, the information on thethree-dimensional watch point may include location information anddistance information in the stereo image. Here, the location informationin the stereo image may include the information on a binocular gazevector.

The communicator 140 may communicate with various types of externalapparatus according to various types of communication method.Specifically, the communicator 140 may communicate with an externalapparatus such as a capturing apparatus, an external server, an externaldisplay apparatus and the like.

Specifically, the communicator 140 may receive a plurality of imagesfrom an external apparatus through a wired method such as an antenna, acable, or a port, or receive a plurality of images through a wirelessmethod such as Wi-Fi and Bluetooth.

In addition, the communicator 140 may transmit the information on thebinocular watch point compensated by an operation of the processor 120and the information on a three-dimensional watch point generated by thecompensated binocular watch point, to an external apparatus.

Meanwhile, FIG. 3 illustrates that the inputter 110 and the communicator140 are an additional configuration, but the electronic apparatus 100may not include a capturing unit or the inputter 110 may include some orall functions of the communicator 140 by a setting.

The display 150 may display the image corresponding to the generatedthree-dimensional watch point. Specifically, the display 150 may furtherdisplay a curser and the like at the location corresponding to thethree-dimensional watch point being watched by a user in thestereoscopic image generated by using a stereo image.

The video processor 160 is an element for processing the contentreceived through the communicator 140 or video data included in thecontent stored in the memory 130 when the display 150 is included in theelectronic apparatus 100. The video processor 160 may perform variousimage processing processes such as decoding, scaling, noise filtering,frame rate conversion, resolution conversion, conversion into astereoscopic image using the stereo image on video data.

In addition, the video processor 160 may process image processing on thevideo data based on the user's gaze detected by the processor 120.Specifically, the video processor 160 may process image processing todisplay a cursor and the like on the received stereoscopic image basedon the three-dimensional watch point generated by the compensatedbinocular watch point.

FIG. 4 is a view illustrating a disparity map according to anembodiment. Specifically, an electronic apparatus may generate adisparity map regarding each of the left eye image and the right eyeimage using the left eye image and the right eye image that constitute astereo image.

Part (a) of FIG. 4 is one of the left eye image and the right eye imagethat constitute the stereo image, and part (b) of FIG. 4 is a disparitymap corresponding to the image illustrated in part (a) of FIG. 4.

Specifically, the disparity map may be expressed as a gray valuecorresponding to a distance. Referring to part (b) of FIG. 4, as thelocation difference between the corresponding points of the left eyeimage and the right eye image is great, the electronic apparatusexpresses the disparity map with a bright gray value, and as thelocation difference of the corresponding points is small, the electronicapparatus expresses the disparity map with a dark gray value whengenerating the disparity map. Accordingly, the electronic apparatus maycalculate the distance information between the object located at aspecific location and a capturing point by using the gray value in aspecific point in the disparity map.

FIG. 5 is a view illustrating a method for compensating a watch pointaccording to an embodiment.

Referring to FIG. 5, the stereo cameras 113 and 114 may generate thestereo images 510 and 520 by capturing the direction corresponding tothe gaze of a user. Specifically, the camera 113 corresponding to theleft eye may generate the left eye image 510, and the camera 114corresponding to the right eye may generate the right eye image 520.Here, the camera 113 corresponding to the left eye and the camera 114corresponding to the right eye may be spaced apart from each other for adistance corresponding to the distance between the left eye and theright eye. In addition, the location difference of the correspondingpoint in the left eye image 510 and the right eye image 520 maycorresponds to the distance between the left eye and the right eye.

The electronic apparatus may detect the left eye watch point (X₁) in theleft eye image 510 and the right eye watch point (X₂) in the right eyeimage 520 using the binocular image. However, if the detected left eyewatch point (X₁) and the detected right eye watch point (X₂) are matchedand the watch point on the three-dimensional space is calculated, theleft eye watch point 11 on the three-dimensional space and the right eyewatch point 12 on the three-dimensional space may not correspond to eachother, and accordingly, the electronic apparatus may not calculate anaccurate watch point on the three-dimensional space.

According thereto, the electronic apparatus may generate the disparitymap 530 regarding the left eye image 510 and the disparity map 540regarding the right eye image 520. In addition, by using the generateddisparity maps 530 and 540, the detected left eye watch point (X₁) andthe detected right eye watch point (X₂) could be compensatedrespectively.

Specifically, the electronic apparatus may calculate the disparity ofthe detected left eye watch point (X₁) and the detected right eye watchpoint (X₂) respectively by using the disparity maps 530 and 540.Referring to FIG. 5, the disparity of the detected left eye watch point(X₁) is far and the disparity of the detected right eye watch point (X₂)is close, and thus the electronic apparatus may determine that the wrongwatch point is detected.

In addition, the electronic apparatus may set the scope in whichcompensation could be performed respectively based on the detected lefteye watch point (X₁) and the detected right eye watch point (X₂), andselect the most appropriate left eye watch point ({circumflex over(X)}₁) and right eye watch point ({circumflex over (X)}₂) within thescope in which compensation could be performed. The operation forcompensating a watch point has been described in the descriptionregarding FIG. 1, and thus an overlapping description will be omitted.

In addition, the electronic apparatus may compensate the watch pointwith the selected left eye watch point ({circumflex over (X)}₁) and theselected right eye watch point ({circumflex over (X)}₂). In addition,the electronic apparatus may generate the watch point 13 on athree-dimensional space using the compensated left eye watch point({circumflex over (X)}₁) and the compensated right eye watch point({circumflex over (X)}₂). Referring to FIG. 5, the watch point on thethree-dimensional space generated using the compensated left eye watchpoint ({circumflex over (X)}₁) is identical to the watch point on thethree-dimensional space generated using the compensated right eye watchpoint ({circumflex over (X)}₂), and thus the electronic apparatus maygenerate the information on the generated watch point on thethree-dimensional space. Here, the generated information may include thevector and distance information of a binocular gaze regarding the watchpoint on the three-dimensional space.

FIG. 6 is a view for describing a watch point detected in a stereoimage. Specifically, part (a) of FIG. 6 is a left eye image 610, andpart (b) of FIG. 6 is a right eye image 620. In addition, FIG. 7 is aview for illustrating a watch point generated by compensating the watchpoint of FIG. 6 according to an embodiment. Specifically, part (a) ofFIG. 7 may be a left eye image 610 and part (b) of FIG. 7 is a right eyeimage 620.

Referring to FIG. 6, the left eye watch point 611 detected from the lefteye image 610 is located on an object. On the contrary, the right eyewatch point 621 detected from the right eye image 620 is located outsidethe object.

The electronic apparatus may generate a disparity map regarding each ofthe left eye image 610 and the right eye image 620, and determine thatthe disparity of the left eye watch point 611 does not correspond to thedisparity of the right eye watch point 621 by using the generateddisparity map. In addition, the electronic apparatus may compensate theleft eye watch point 611 and the right eye watch point 621 using thedisparity map.

Specifically, based on the left eye watch point 611 and the right eyewatch point 621, the electronic apparatus may set the area in which acompensation may be performed respectively, and may select the mostappropriate left eye watch point and right eye watch point so that acompensation distance is minimum in the area in which compensation maybe performed and that the difference between the disparity of eachcompensated watch point is minimum.

The electronic apparatus may compensate the left eye watch point and theright eye watch point with the selected left eye watch point and theright eye watch point. As a result of performing the compensationaccording to an embodiment, as illustrated in FIG. 7, the compensatedleft eye watch point 613 in the left eye image 610 may be located on anobject, and the compensated right eye watch point 622 in the right eyeimage 620 also may be located on an object.

As in the above, in a state in which the horizontal deviation of theleft eye image 610 and the right eye image 620 is compensated, theelectronic apparatus may perform compensation so that the locationdifference between the compensated left eye watch point 613 and thecompensated right eye watch point 622 is minimized, and that thedisparity difference between the compensated left eye watch point 613and the compensated right eye watch point 622 is minimized.

FIG. 8 is a flow chart illustrating a method for compensating a watchpoint according to an embodiment.

First, the electronic apparatus may receive a binocular image which is acaptured image of both eyes of a user and a stereo image which is animage of a direction corresponding to a gaze of the user captured atlocations spaced apart from each other, in S810. Specifically, theelectronic apparatus may capture the binocular image and the stereoimage using a plurality of included cameras. Meanwhile, if a camera isnot included in the electronic apparatus, at least one of the binocularimage and the stereo image may be received through an externalapparatus.

Next, the electronic apparatus may detect the watch point of a userusing the binocular image in S820. Specifically, the electronicapparatus may detect the watch point of the user in the stereo imageusing the binocular image.

In addition, the electronic apparatus may generate a disparity map inthe input stereo image in S830. Specifically, the electronic apparatusmay generate the disparity map regarding the left eye image and theright eye image respectively using the left eye image and the right eyeimage constituting the stereo image. Here, the electronic apparatus maygenerate a disparity map using the difference in the locations of acorresponding points in the left eye image and the right eye image.

Next, the electronic apparatus may compensate the detected watch pointusing the generated disparity map in S840. Specifically, the electronicapparatus may compensate the left eye watch point detected from the lefteye image and the right eye watch point detected from the right eyeimage by using the disparity map generated regarding the left eye imageand the disparity map generated regarding the right eye image. Here, theelectronic apparatus may compensate the left eye watch point and theright eye watch point at one time.

As described above, according to compensating the left eye watch pointand the right eye watch point using a disparity map, further accuratewatch point of a user may be detected in a three-dimensional space.

Various embodiments described above may be implemented in a recordingmedium that may be read by a computer or a similar apparatus to thecomputer by using software, hardware, or a combination thereof.According to the hardware embodiment, exemplary embodiments that aredescribed in the present disclosure may be implemented by using at leastone selected from Application Specific Integrated Circuits (ASICs),Digital Signal Processors (DSPs), Digital Signal Processing Devices(DSPDs), Programmable Logic Devices (PLDs), Field Programmable GateArrays (FPGAs), processors, controllers, micro-controllers,microprocessors, electrical units for performing other functions. Insome cases, exemplary embodiments that are described in the presentdisclosure may be embodied as the processor 120 itself. In a softwareembodiment, various embodiments described in the present disclosure suchas a procedure and a function may be implemented as separate softwaremodules. The software modules may respectively perform one or morefunctions and operations described in the exemplary embodiments.

Methods of controlling an electronic apparatus according to variousexemplary embodiments may be stored in a non-transitory readable medium.The non-transitory readable medium may be installed and used in variousdevices.

The non-transitory computer readable medium is not a medium that storesdata for a short moment such as a register, a cache and a memory and thelike, but a medium that stores data semi-permanently and which isreadable by an apparatus. Specifically, programs for performing theabove-described various methods can be stored in a non-transitorycomputer readable medium such as a CD, a DVD, a hard disk, a Blu-raydisk, universal serial bus (USB), a memory card, ROM, or the like, andcan be provided.

Although various example embodiments of the present disclosure have beenillustrated and described hereinabove, the present disclosure is notlimited to the above-mentioned example embodiments, but may be variouslymodified by those skilled in the art to which the present disclosurepertains without departing from the scope and spirit of the disclosureas set forth in the accompanying claims. These modifications should alsobe understood to fall within the scope of the present disclosure.

What is claimed is:
 1. An electronic apparatus comprising: an inputterconfigured to receive a binocular image which is a captured image ofboth eyes of a user and a stereo image which is an image of a directioncorresponding to a gaze of the user captured at locations spaced apartfrom each other; and a processor configured to: detect a watch point ofa user in the stereo image by using the binocular image, obtain adisparity map in the input stereo image, and compensate the detectedwatch point using the obtained disparity map.
 2. The electronicapparatus as claimed in claim 1, wherein the stereo image includes afirst image corresponding to a left eye of the user and a second imagecorresponding to a right eye of the user, and wherein the processor isfurther configured to: detect a left eye watch point of the user in thefirst image and a right eye watch point of the user in the second imageby using the binocular image, obtain a first disparity map and a seconddisparity map by performing stereo matching of the first image and thesecond image, and compensate the detected left eye watch point and thedetected right eye watch point by using the first disparity map and thesecond disparity map.
 3. The electronic apparatus as claimed in claim 2,wherein the processor is further configured to compensate the detectedleft eye watch point and the detected right eye watch point using thefirst disparity map and the second disparity map to minimize adifference between a disparity of the left eye watch point and adisparity of the right eye watch point.
 4. The electronic apparatus asclaimed in claim 2, wherein the processor is further configured tocompensate the detected left eye watch point and the detected right eyewatch point to minimize a difference between a location of the left eyewatch point and a location of the right eye watch point in a state inwhich horizontal deviation of the first image and the second image iscompensated.
 5. The electronic apparatus as claimed in claim 2, whereinthe processor is further configured to: compensate vertical deviation ofthe first image and the second image, and perform stereo matching usingthe compensated first image and the compensated second image.
 6. Theelectronic apparatus as claimed in claim 1, wherein the processor isfurther configured to obtain a three-dimensional watch point using thecompensated left eye watch point and the compensated right eye watchpoint.
 7. The electronic apparatus as claimed in claim 1, wherein theprocessor is further configured to obtain vector and distanceinformation of a binocular gaze using the compensated left eye watchpoint and the compensated right eye watch point.
 8. The electronicapparatus as claimed in claim 1, wherein the inputter comprises: aninfrared camera configured to captures both eyes of the user; and astereo camera configured to captures a direction corresponding to a gazeof the user at locations spaced apart from each other.
 9. A method forcontrolling an electronic apparatus comprising: receiving a binocularimage which is a captured image of both eyes of a user and a stereoimage which is an image of a direction corresponding to a gaze of theuser captured at locations spaced apart from each other; detecting awatch point of a user in the stereo image by using the binocular image;obtaining a disparity map in the stereo image; and compensating thedetected watch point using the obtained disparity map.
 10. Thecontrolling method as claimed in claim 9, wherein the stereo imageincludes a first image corresponding to a left eye of the user and asecond image corresponding to a right eye of the user, wherein thedetecting the watch point comprises detecting a left eye watch point ofa user in the first image and a right eye watch point in the secondimage by using the binocular image, wherein the obtaining comprisesobtaining a first disparity map and a second disparity map by performingstereo matching of the first image and the second image, and wherein thecompensating comprises compensating the detected left eye watch pointand the detected right eye watch point by using the first disparity mapand the second disparity map.
 11. The controlling method as claimed inclaim 10, wherein the compensating comprises compensating the detectedleft eye watch point and the detected right eye watch point using thefirst disparity map and the second disparity map to minimize adifference between a disparity of the left eye watch point and adisparity of the right eye watch point.
 12. The controlling method asclaimed in claim 10, wherein the compensating comprises compensating thedetected left eye watch point and the detected right eye watch point tominimize a difference between a location of the left eye watch point anda location of the right eye watch point in a state in which horizontaldeviation of the first image and the second image is compensated. 13.The controlling method as claimed in claim 10, further comprising:compensating vertical deviation of the first image and the second image;wherein the obtaining comprises performing stereo matching using thecompensated first image and the compensated second image.
 14. Thecontrolling method as claimed in claim 9, further comprising: obtaininga three dimensional watch point using the compensated left eye watchpoint and the compensated right eye watch point.
 15. The controllingmethod as claimed in claim 9, further comprising: obtaining vector anddistance information of a binocular gaze using the compensated left eyewatch point and the compensated right eye watch point.
 16. Thecontrolling method as claimed in claim 9, wherein the receivingcomprises capturing both eyes of the user using an infrared camera, andcapturing a direction corresponding to a gaze of the user at locationsspaced apart from each other.
 17. A non-transitory computer readablerecording medium which includes a program to execute a method forcontrolling an electronic apparatus, wherein the controlling methodcomprises: receiving a binocular image which is a captured image of botheyes of a user and a stereo image which is an image of a directioncorresponding to a gaze of the user captured at locations spaced apartfrom each other; detecting a watch point of a user in the stereo imageby using the binocular image; obtaining a disparity map in the stereoimage; and compensating the detected watch point using the obtaineddisparity map.